<html>
<head><meta charset="utf-8"><title>Speculative CI? · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html">Speculative CI?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="200519964"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200519964" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> trevyn <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200519964">(Jun 11 2020 at 09:02)</a>:</h4>
<p>This bors queue is pretty deep. What is the current thinking on how to manage this going forward? Are we limited on CI resources, or is Microsoft donating as much as we can use? Specifically, it would be cool if bors did speculative runs, i.e. assume that the current CI run succeeds, pick the next PR as well and merge it on top, and then run that speculatively in parallel, so that if the first succeeds, two PRs can potentially get merged in quick succession.</p>



<a name="200520711"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200520711" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200520711">(Jun 11 2020 at 09:11)</a>:</h4>
<p>We are already doing rollups to improve throughput. Rollups combine multiple PR's together to test them at once. Doing speculative CI will waste a lot of resources because about half of the time CI runs fail, as can be seen at <a href="https://dev.azure.com/rust-lang/rust/_build?definitionId=25">https://dev.azure.com/rust-lang/rust/_build?definitionId=25</a>. (runs where one jobs fails are automatically canceled) Also I think Microsoft has reserved specific machines for the Rust CI.</p>



<a name="200521430"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200521430" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> trevyn <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200521430">(Jun 11 2020 at 09:19)</a>:</h4>
<p>Understood that it is less efficient and that rollups are a current strategy; I want to understand what Microsoft is willing to contribute going forward. As a new contributor, it is somewhat discouraging to see my PR take a week to go from approval to CI run, and I can't help but think that this has an impact on Rust iteration speed in general.</p>



<a name="200521928"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200521928" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> marmeladema <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200521928">(Jun 11 2020 at 09:25)</a>:</h4>
<p>I think the infra team is trying github actions to replace Azure, and build seems way faster but its not reliable enough right now iirc.</p>



<a name="200522729"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200522729" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> trevyn <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200522729">(Jun 11 2020 at 09:34)</a>:</h4>
<p>Thanks <span class="user-mention" data-user-id="281572">@marmeladema</span> ! Can you point me to where this is being discussed?</p>



<a name="200522813"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200522813" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200522813">(Jun 11 2020 at 09:35)</a>:</h4>
<p>This is the issue for the reliability problem: <a href="https://github.com/rust-lang/rust/issues/71988">https://github.com/rust-lang/rust/issues/71988</a></p>



<a name="200522948"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200522948" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200522948">(Jun 11 2020 at 09:37)</a>:</h4>
<p><a href="https://github.com/rust-lang/rust/issues?q=is%3Aissue+is%3Aopen+label%3AA-github-actions">https://github.com/rust-lang/rust/issues?q=is%3Aissue+is%3Aopen+label%3AA-github-actions</a> is the list of all github actions related issues.</p>



<a name="200523392"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200523392" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> trevyn <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200523392">(Jun 11 2020 at 09:43)</a>:</h4>
<p>Cool, thank you! Have we reached out to our contacts at Google and Amazon regarding their willingness to contribute to CI runs?</p>



<a name="200523506"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200523506" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> trevyn <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200523506">(Jun 11 2020 at 09:44)</a>:</h4>
<p>Specifically also looping in the teams at Google that we know use Rust... ;)</p>



<a name="200524560"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200524560" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> DPC <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200524560">(Jun 11 2020 at 09:58)</a>:</h4>
<p>Just to add that the queue is deep right now because we had an issue with wasm target yesterday so had to close the queue for around 6-8 hours in the evening. Normally the queue looks much better than this. <br>
Not part of the infra team, but from what I know there are certain discussions that are happening and we will get updates regarding it.</p>



<a name="200533333"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200533333" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200533333">(Jun 11 2020 at 11:50)</a>:</h4>
<p>A while back I suggested incremental retesting, where only tests affected by the changed code would be run. So for example, if you only modify rustdoc files, you don't have to rerun all the thousands of UI tests. That was rejected because there was worried about making CI less dependable and more magic: <a href="https://discord.com/channels/442252698964721669/443148319431065610/672615346997428249">https://discord.com/channels/442252698964721669/443148319431065610/672615346997428249</a></p>



<a name="200534454"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200534454" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200534454">(Jun 11 2020 at 12:02)</a>:</h4>
<p>For PR tests I think CI being less dependable would be less of a problem than for the final tests before merge. It would still have the problem of being more complex to maintain though.</p>



<a name="200535620"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200535620" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> DPC <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200535620">(Jun 11 2020 at 12:15)</a>:</h4>
<p><span class="user-mention" data-user-id="232545">@Joshua Nelson</span> it is difficult given that certain changes could cause regression in other parts. It doesn't happen that often but it has occurred in the past that a test has failed because it depends on the changed code transitively instead of directly.</p>



<a name="200535974"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200535974" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200535974">(Jun 11 2020 at 12:18)</a>:</h4>
<blockquote>
<p>it has occurred in the past that a test has failed because it depends on the changed code transitively instead of directly.</p>
</blockquote>
<p>My idea took care of that by calculating <em>all</em> dependencies recursively ... It makes me a little sad people weren't listening. That does take the scope from "add code to x.py" to "write an independently maintained project" but I was offering to do all the work</p>



<a name="200541776"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200541776" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200541776">(Jun 11 2020 at 13:12)</a>:</h4>
<p><span class="user-mention" data-user-id="232545">@Joshua Nelson</span> <a href="#narrow/stream/242791-t-infra/topic/Speculative.20CI.3F/near/200535974">said</a>:</p>
<blockquote>
<p>My idea took care of that by calculating <em>all</em> dependencies recursively ... It makes me a little sad people weren't listening. That does take the scope from "add code to x.py" to "write an independently maintained project" but I was offering to do all the work</p>
</blockquote>
<p>to be honest running the actual tests doesn't take much long, and it's something that can be fixed by throwing more cores at the problem, while last time I checked most of the time was spent compiling the compiler (or other tooling in dist builders) instead of running tests</p>



<a name="200542060"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200542060" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200542060">(Jun 11 2020 at 13:14)</a>:</h4>
<blockquote>
<p>This bors queue is pretty deep. What is the current thinking on how to manage this going forward? Are we limited on CI resources, or is Microsoft donating as much as we can use?</p>
</blockquote>
<p><span class="user-mention" data-user-id="278163">@trevyn</span> currently we're limited by the hosted builders on azure pipelines, that are only available with 2 cores -- we're already working with microsoft to migrate to everything to github actions, which has a custom 8-core hosted VMs pool dedicated just for the rust project</p>



<a name="200542229"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200542229" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200542229">(Jun 11 2020 at 13:16)</a>:</h4>
<p>progress on that is slower than I'd like (we're blocked on macOS builders being really flaky, and github is investigating into it), but hopefully it'll be ready soon</p>



<a name="200542252"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200542252" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200542252">(Jun 11 2020 at 13:16)</a>:</h4>
<p>with how many builders we use, speculative execution is probably going to be way more wasteful than just making single builds faster and doing rollups</p>



<a name="200550352"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200550352" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> marmeladema <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200550352">(Jun 11 2020 at 14:18)</a>:</h4>
<p>I also wondered, but maybe that's impossible or totally none sense, but could the some parts like the standard library be hosted in another repo and have a lighter build / testing process? Lik</p>



<a name="200563156"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200563156" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200563156">(Jun 11 2020 at 15:49)</a>:</h4>
<p><span class="user-mention" data-user-id="281572">@marmeladema</span> that's what we're already doing with some of the tools and the toolstate system, but I don't think it's suitable for the stdlib</p>



<a name="200563329"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200563329" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200563329">(Jun 11 2020 at 15:50)</a>:</h4>
<p>the problem with that tooling is that sometimes changes to the compiler are made that break one of the submodules, and that submodule stops being available on nightly until a separate PR is sent to the submodule and another PR bumps the submodule on the main repo</p>



<a name="200563397"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200563397" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200563397">(Jun 11 2020 at 15:50)</a>:</h4>
<p>we don't care <em>too much</em> if rustfmt or clippy are missing for a nightly or two, but not having the stdlib available would make the nightly useless</p>



<a name="200563464"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200563464" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200563464">(Jun 11 2020 at 15:51)</a>:</h4>
<p>also I know there are some efforts to scale down the number of submodules we manage, as the setup is a bit problematic for contributors</p>



<a name="200565298"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Speculative%20CI%3F/near/200565298" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> marmeladema <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Speculative.20CI.3F.html#200565298">(Jun 11 2020 at 16:02)</a>:</h4>
<p>I thought subtree kind of solve that issue? I don't know much about substree though but i thought maybe the standard library could use the same feature.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>